Chromakeying system

ABSTRACT

A chromokeying system includes a store that contains a representation of a known coded chromokeying pattern. The system further includes a frame grabber that grabs a video image including at least a portion of the known coded chromokeying pattern and a foreground subject. A perspective transformation computation unit computes a perspective transformation by comparing the video image of the portion of the coded chromokeying pattern with the known chromokeying pattern. The system may further include a background generation unit that receives a background image and applies the perspective transformation to the background image to produce a transformed background. The portion of the coded chromokeying pattern in the video image can then be replaced by the transformed background to create a composite image.

FIELD OF THE INVENTION

The present invention relates to methods and apparatus for studioimaging including chromakeying.

BACKGROUND OF THE INVENTION

Background chromakey panels are commonly used in the art of televisionand video production in order to facilitate merging of a virtualbackground with foreground figures. The foreground figures are imaged onthe chromakey panel background, and the known color of the chromakeypanel background is detected and replaced by the virtual background. Oneproblem associated with such productions is that changes in theforeground image, due to camera displacement, pan, tilt and zoom in andout, are not automatically applied to the background scene.

One system designed to solve this problem is the "Virtual Studio"developed by Ultimatte. The "Virtual Studio" requires a special memoryhead pedestal which measures the camera's pan and tilt angles directlyfrom the camera's mount. A disadvantage of this approach is that inorder to obtain a complete perspective transformation for the backgroundimage, the zoom state of the lens must be read and the instantaneouslocation of the camera must be determined. Also, this approach requiresthat each and every camera mount be replaced by a memory head pedestal,resulting in an expensive system and complicated operation if a largenumber of cameras are employed. Also, the accuracy of this method isinadequate for seamless foreground/background merging.

Conventional edge detection methods which are based on smoothing andfinding directional derivates are described in the followingpublication, the disclosure of which is hereby incorporated byreference:

Canny, J., "A computational approach to edge detection", IEEE Trans. onPAMI, PAMI-8, No. 6, 1986, 679-697.

Conventional edge detection methods which are based on Hough transformsare surveyed in the following publication, the disclosure of which ishereby incorporated by reference:

J. Illingworth and J. Kitler, "A survey of the Hough transform", CVGIP,44, 1988, 87-116.

FIG. BA-1 is a simplified block diagram of a state of the artchromakeying system.

SUMMARY OF THE INVENTION

The present invention seeks to provide improved methods and apparatusfor chromakeying.

There is thus provided in accordance with a preferred embodiment of thepresent invention a chromakeying panel for use in conjunction with achromakeying system having a color keying spectrum, the panel includinga plurality of first locations bearing a first color within the colorkeying spectrum, and a plurality of second locations bearing a secondcolor within the color keying spectrum.

Further in accordance with a preferred embodiment of the presentinvention the first color and the second color are of the same hue andof different intensities.

There is additionally provided in accordance with a preferred embodimentof the present invention a chromakeying system including a frame grabberoperative to grab an image of a foreground image including at least onenonuniform chromakeying panel having a known pattern and a subject, aperspective transformation computation unit operative to compute aperspective transformation by comparing the image of at least oneportion of the captured panel with the known pattern, and a backgroundgeneration unit operative to receive a background image, to apply theperspective transformation thereto and to replace the captured portionof at least one panel with the transformed background, thereby togenerate a composite image.

Still further in accordance with a preferred embodiment of the presentinvention the system also includes a 3D background image sourceoperative to provide the background generation unit with a 3Drepresentation of a background scene.

Additionally in accordance with a preferred embodiment of the presentinvention the system also includes a background image sequence sourceoperative to provide the background generation unit with a temporalsequence of background scenes and wherein the perspective transformationcomputation unit is operative in real time relative to the pace of thetemporal sequence.

Also in accordance with a preferred embodiment of the present inventionthe background image sequence source includes a video clip source andwherein the perspective transformation computation unit is operative invideo real time.

Further in accordance with a preferred embodiment of the presentinvention the system also includes a graphically animated backgroundimage source operative to provide the background generation unit with agraphically animated background scene.

Still further in accordance with a preferred embodiment of the presentinvention the system also includes a photographed background imagesource operative to provide the background generation unit with aphotographed background scene.

Additionally in accordance with a preferred embodiment of the presentinvention the system includes a user interface communicating with thebackground generation unit and operative to determine, for individualvirtual object locations within the background image, whether thevirtual object location is in front of the subject or in back of thesubject and wherein the background generation unit is operative tosuperimpose the subject over back virtual object locations and tosuperimpose front virtual object locations over the subject.

Further in accordance with a preferred embodiment of the presentinvention each individual virtual object location includes a singlepixel within the background image.

Still further in accordance with a preferred embodiment of the presentinvention the system includes a rangefinder boresighted to the cameraimaging at least one chromakey panel and the subject and operative tomeasure the range of a plurality of locations within the camera's fieldof view and wherein the background generation unit is operative to mergeoverlaying background image locations and subject locations such thatthe overlaying location having a smaller range is included in thecomposite image.

Further in accordance with a preferred embodiment of the presentinvention the system also includes a plurality of cameras imaging atleast one chromakey panel and the subject.

Still further in accordance with a preferred embodiment of the presentinvention the at least one chromakey panel includes a plurality ofchromakey panels.

Additionally in accordance with a preferred embodiment of the presentinvention the background generator is operative to replace chromakeypanel pixels with a transformed background also in the case that thecamera's momentary field of view is larger than the chromakey panel'sfield of view.

There is further provided in accordance with a preferred embodiment ofthe present invention a chromakeying method for use in conjunction witha chromakeying system having a color keying spectrum, the methodincluding providing a chromakeying panel including a plurality of firstlocations bearing a first color within the color keying spectrum and aplurality of second locations bearing a second color within the colorkeying spectrum, and imaging a foreground image including the panel anda subject.

There is additionally provided in accordance with a preferred embodimentof the present invention a chromakeying method including grabbing animage of a foreground image including at least one nonuniformchromakeying panel having a known pattern and a subject, computing aperspective transformation by comparing the image of at least one panelwith the known pattern, and receiving a background image, applying theperspective transformation thereto and replacing at least one panel withthe transformed background, thereby to generate a composite image.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be understood and appreciated from thefollowing detailed description, taken in conjunction with the drawingsin which:

FIG. BA-1 is a simplified block diagram of a prior art chromakeyingsystem;

FIG. 1 is a simplified block diagram of a chromakeying systemconstructed and operative in accordance with a preferred embodiment ofthe present invention;

FIG. 2 is a simplified flowchart of a preferred method of operation forthe perspective transformation computation unit, perspectivetransforming unit and chromakeying unit of FIG. 1;

FIG. 3 is a simplified block diagram of an application for the apparatusof FIG. 1, in a multi-camera studio;

FIG. 4 is a modification of the apparatus of FIG. 1 in which arange-finder with spatial resolution, boresighted to the camera, is usedto obtain range data for foreground objects;

FIG. 5 is a diagram of a portion of a pattern for a chromakey panelconstructed and operative in accordance with a preferred embodiment ofthe present invention; and

FIG. 6 illustrates the effect of camera displacement on the panelportion of FIG. 5;

FIG. 7 illustrates the effect of zoom on the panel portion of FIG. 5;

FIG. 8 illustrates the effect of roll on the panel portion of FIG. 5;

FIG. 9 illustrates the effect of a combination of camera tilt anddisplacement on the panel portion of FIG. 5; and

FIG. 10 illustrates the effect of a combination of camera pan anddisplacement on the panel portion of FIG. 5.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

FIG. BA-1 illustrates a prior art chromakeying system comprising achromakeyer 250 and background image storage system 260. The prior artsystem 260 includes a background data bank for generating a "virtual"background scene.

Reference is now made to FIG. 1 which illustrates an improvedchromakeying system which is constructed and operative in accordancewith a preferred embodiment of the present invention. The apparatus ofFIG. 1 includes a conventional camera 10 mounted on a conventionalpedestal 14. The camera 10 images a chromakey panel 20 which is abackdrop for a subject or foreground object 30 such as a human figure.The chromakey panel typically comprises a colored pattern 40 having thefollowing characteristics:

a. the pattern includes two or more shades both or all fall within thechromakeying spectrum of the apparatus,

b. The pattern allows the imaged pattern to be warped to the knownactual pattern, and

c. The pattern allows perspective distortions of the actual patternduring imaging, due to zoom, linear displacement and angulardisplacement of the camera 10, to be detected and applied to a virtualbackground.

A sample pattern having the above characteristics is described in moredetail below with reference to FIG. 5. The apparatus of FIG. 1 alsoincludes a chromakeyer 50 and a background generator 60. The chromakeyer50 is operative to identify the areas within the camera's output imagewhich correspond to the chromakey panel and to replace these areas witha virtual background supplied by background generator 60.

A particular feature of the present invention is that the backgroundsupplied by background generator 60 is in perspective with respect tothe current camera's orientation and preferably has integratedtherewithin other features of the foreground image such as blur, shadingand lighting non-uniformity.

The background generator 60 includes a conventional frame grabber 70which grabs the foreground image from the camera 10 and feeds it to aperspective transformation computation unit 80. The perspectivetransformation computation unit 80 is operative to analyze theperspective transformation of the chromakey panel pattern in the grabbedforeground image, by comparison with the known chromakey panel pattern.Preferably, other imaging features which affect the panel pattern arealso analyzed, such as blur, shading and lighting non-uniformity.

Perspective transformation computation unit 80 feeds the transformationparameters to a perspective transformation unit 90 which applies theperspective transformation to the desired virtual background image. Thevirtual background may be supplied from any suitable source such as ahard disk 100, an analog video recorder 110 (disk, cassette or tape)associated with a frame grabber 126 or from an animation computer 132.

The output of the system is an image 142 which may be broadcasted orotherwise displayed to an audience which includes a virtual backgroundhaving the foreground image perspective and other realistic effects suchas blur, lighting and shading. It is appreciated that the perspectiveeffect and the blur and lighting effects in image 142 vary over time inexact correspondence with the variation over time of the same effects inthe original foreground image.

Reference is now made to FIG. 2 which is a simplified flowchart of apreferred method of operation for the perspective transformationcomputation unit, perspective transforming unit and chromakeying unit ofFIG. 1.

The method of FIG. 2 preferably comprises the following steps:

STEP 120: The frame grabber grabs and digitizes the output of thecamera, one field at a time or one frame at a time. It is preferable toprocess field by field due to the camera's angular motion.

STEP 122: If the current grabbed field belongs to the first frame of thesession, proceed to step 140. Otherwise proceed to step 124, i.e. if thecurrent grabbed field is a subsequent frame of the session:

STEP 124: Search for the pattern portion analyzed in the previous field,in a window surrounding the corresponding location in the previousfield. Alternatively, the search may be conducted inside a windowsurrounding a new location whose distance from the location in theprevious field is determined by predicting the camera's angular speedaccording to the detected pattern locations in the last consecutivefields.

To detect the pattern edges, conventional edge detection methods may beemployed, such as either of the following:

Canny, J., "A computational approach to edge detection", IEEE Trans. onPAMI, PAMI-8, No. 6, 1986, 679-697; or

J. Illingworth and J. Kitler, "A survey of the Hough transform", CVGIP,44, 1988, 87-116.

STEP 130: Decision: Does the current field belong to a "new" frame, i.e.does it directly follow a new cut in which the camera has been switched,or is the current frame a consecutive frame, i.e. an additional framefrom the same camera location as the previous frame. The processingafter step 130 is on a field by field basis.

If the frame is a new frame, then:

STEP 140: Optionally, find the panel within the camera output image bysearching for an area of pixels whose colors fall within the chromakeyspectrum. Pattern edge detection may then be performed from thisstarting point rather than from an arbitrary starting point.

STEP 150: Perform edge detection on the captured panel portion,preferably with single pixel or subpixel accuracy, using conventionaledge detection methods such as those described in the above-referencedCanny and Illingworth publications. The pattern lines are thenconstructed using the detected edges. Edge detection is typicallyperformed along each of the horizontal and vertical dimensions at leastuntil two edges have been found along one of the dimensions and threeedges along the other of the dimensions and altogether five lines havebeen defined. Finding two lines in one dimension and three lines in theother results in six vertices connecting these lines which aresubsequently matched to the known panel pattern as described below withreference to step 160.

STEP 154: Within the detected edges, distinguish the pattern fromnon-pattern features which obscure the pattern, such as shadows, lensdistortions, and lighting nonuniformities. For example, the patternfeatures may be distinguished by their straight-line segments, as in thepattern of FIG. 5, because the non-pattern features usually have curvededges. Also, non-pattern edges such as those due to lightingnonuniformities are often smooth whereas the pattern edges are sharp inthe direction perpendicular to camera motion. The locations of thenon-pattern edges are stored in order to allow subsequent correction ofthe virtual background image. The non-pattern edges are ignored in thesubsequent analysis.

Also stored is an indication of the color intensity map inside theshadow edges or within lighting nonuniformity contours or inside anycontour isolating regions of lens abberation, such as vignetting.

STEP 158: Optionally, quantify the blur introduced into the pattern bythe camera's scanning during the imaging field time. The blur level at aplurality of locations within the image is stored so that the same blurlevel can be virtually applied to the virtual background. Typically, theblur level is stored as a "map" which indicates the blur levelassociated with a grid of pixels located on the detected lines. The blurlevels at other image points may be interpolated from this map.

STEP 160: For at least four locations (typically vertices) in the imagedpanel portion, find the same number of matching locations (vertices) inthe original panel pattern. For example, locations may be matched on thebasis of unique ratios between line segments, as described below indetail with reference to FIG. 5. For localization, two line segments arenormally required in one direction, so three detected lines are normallyneeded in the perpendicular dimension.

At least four imaged panel locations (x_(i),Y_(i)) are typically matchedto a corresponding number of original panel pattern locations(X_(i),Y_(i)).

STEP 164: Use the paired locations to compute the linear displacement,angular displacement and zoom change of the imaging camera with respectto the previous orientation or with respect to the original cameraorientation.

The computations may be based on the following equations 1-7 forperspective transformation, where:

X_(O), Y_(O), Z_(O) =the linear displacement of the imaging camera;

The angular displacement parameters are α (pan), β (tilt), and gamma(roll), respectively;

f=the zoom of the imaging camera;

R_(x) (α)=Rotation about the X-axis;

R_(y) (β)=Rotation about the Y-axis; and

R_(z) ()=Rotation about the Z-axis. ##EQU1##

STEP 170: Receive the background image from the framegrabber 70 or harddisk 100 or animation computer 132 of FIG. 1 and transform thebackground image so as to reproduce the linear displacement, angulardisplacement and focal length change of the imaging camera. Everybackground pixel i whose center is in (X_(i), Y_(i)) coordinates, is nowtransformed to location (x_(i), y_(i)) using equations (1-7). Typically,this operation is performed inversely: for each destination pixel iwhose center is at (x_(i), y_(i)) the source location (X_(i), Y_(i)) iscomputed using the inverse of equations 6 and 7. The destination pixelvalue is then computed by weighting the source neighboring pixels usingnearest neighbors, bi-linear approximations or tri-linearapproximations.

STEP 174: Optionally, the transformed image is corrected so as tovirtually apply the stored blur, shadow and lighting nonuniformitiesdata to the transformed image, thereby to emulate, within thebackground, the same type of effects which are occurring in theforeground.

STEP 180: The composite image is generated by replacing the chromakeypanel pixels in the foreground image with the corresponding pixels ofthe transformed and corrected background image.

FIG. 3 is a simplified block diagram of an application for the apparatusof FIG. 1, in a multi-camera studio which includes an array of n cameras10 and a switcher 150 which enables a producer to switch from one camerato another. It is appreciated that only one background generator 60 isrequired for the array of cameras 10.

FIG. 4 is a modification of the apparatus of FIG. 1 which generates aperspectively transformed 3D background. A storage medium 260, such as acomputer hard disk array, is provided which stores a virtual3background. A rangefinder 270 with spatial resolution, typically apulsed laser transmitter/detector module, is boresighted to the imagingcamera and scans the camera's field of view measuring the range to eachvisible pixel or a group of pixels using pulse "time of flight".Range-finder 270 is associated with the camera 10 and pedestal 14 and isoperative to detect the range to each pixel or group of pixels visiblein the captured image. A laser controller and detector's signalprocessor 274 is associated with rangefinder 270 and is operative tocontrol laser pulse timing and to extract momentary range from thereceived echoes.

The chromakeyer 50 is replaced by a chromakeyer 280 which is augmentedwith an occlusion analysis function. Chromakey panel pixels are replacedwith the transformed and preferably corrected virtual background. Foreach background pixel the range measured with the rangefinder iscompared to the Z-value of the corresponding background pixel. If the Zvalue is smaller than the range, the Chromakey displays the backgroundpixel.

Reference is now made to FIG. 5 which is a diagram of a portion of apattern for a chromakey panel constructed and operative in accordancewith a preferred embodiment of the present invention.

The pattern portion of FIG. 5 typically comprises an uneven checkerboardof quadrilaterals having two alternating shades within the chromakeyspectrum. The edges of the quadrilaterals are arranged along twodimensions, which may be perpendicular, such as the horizontal andvertical dimensions, but need not necessarily be so. The pattern portionof FIG. 5 preferably has the following attribute for each dimension d:

For every vertex 610, the ratio between the length of the line segment620 on one side thereof, along the dimension d, and the length of theline segment 630 on the other side thereof, along the dimension d, isunique. In other words, there is no other vertex whose adjacent linesegments along the dimension d or along both directions have the sameratio between their lengths. This attribute allows vertices, whendetected, to be localized by computing the above ratio and comparing itto the known ratios associated with the various vertices.

For example, in the illustrated embodiment, the ratio between thelengths of segments 620 and 630 is 2 and none of the vertices of FIG. 5,other than vertex 610, have the same ratio.

Preferably, the pattern also has the following pair of attributes:

a. At least two pattern vertices in one direction and three in the otherdirection are included in the minimal field of view (maximummagnification) of the camera,

b. In the maximal field of view (minimal magnification) of the camera,the imaging camera resolution is sufficient to allow adjacent edges tobe separated by conventional edge detection methods.

A suitable panel may be approximately 3 m×3 m in size and may have anuneven checkerboard pattern, such that the sequence of ratios betweenadjacent line segments along one or each dimension is a permutation ofthe following numbers: 1.01, 1.03, 1.05, 1.07, 1.09, 1.11, 1.13, 1.15,1.17, 1.19, . . . The largest line segment may be determined bycomputation such that at least six pattern vertices are included in theminimal field of view (maximum magnification) of the camera. Thepermutation of the above sequence of numbers is preferably such thatsmall ratios are adjacent to large ratios. For example, the first 6numbers may be: 1.01, 1.19, 1.03, 1.17, 1.05, 1.15, and so on.

The field of view of the camera can, if desired, exceed the size of thechromakey panel.

Reference is also made to FIGS. 6-10 which illustrate the effects ofcamera displacement; zoom; roll; a combination of tilt and displacement;and a combination of pan and displacement, respectively, on the panelportion of FIG. 5. In the present specification and claims, roll, tiltand pan are the angular displacements about the Z axis, X axis and Yaxis, respectively, where the panel plane is the X-Y plane.

In FIG. 9, the camera is first tilted and then displaced in parallel tothe panel plane to capture the same panel portions. In FIG. 10, thecamera is first panned and then displaced in parallel to the panel planeto capture the same panel portions.

The abbreviations "FG" and "BG" are used for "foreground" and"background", respectively. The term "foreground" refers to the subject(such as the protagonists of a talk show) and the chromakey panel whichis the backdrop. The term "background" refers to the image whichreplaces the chromakey panel, such as an image of an outdoor scene.

It is appreciated that the software components of the present inventionmay, if desired, be implemented in ROM (read-only memory) form. Thesoftware components may, generally, be implemented in hardware, ifdesired, using conventional techniques.

It is appreciated that various features of the invention which are, forclarity, described in the contexts of separate embodiments may also beprovided in combination in a single embodiment. Conversely, variousfeatures of the invention which are, for brevity, described in thecontext of a single embodiment may also be provided separately or in anysuitable subcombination.

It will be appreciated by persons skilled in the art that the presentinvention is not limited to what has been particularly shown anddescribed hereinabove. Rather, the scope of the present invention isdefined only by the claims that follow:

We claim:
 1. A chromakeying system comprising:a store operative to storea representation of a known coded chromakeying pattern; a frame grabberoperative to grab a video image including at least a portion of thecoded chromakeying pattern and a foreground subject; a perspectivetransformation computation unit operative to compute a perspectivetransformation by comparing the video image of the portion of the codedchromakeying pattern with the known coded chromakeying pattern; and abackground generation unit operative to receive a background image,apply the perspective transformation thereto to produce a transformedbackground and replace the portion of the coded chromakeying pattern inthe video image with the transformed background and, thereby generate acomposite image.
 2. The chromakeying system of claim 1, furthercomprising a 3D background image source operative to provide thebackground generation unit with the background image, the backgroundimage being a 3D representation of a background scene.
 3. Thechromakeying system of claim 1 further comprising a background imagesequence source operative to provide the background generation unit witha temporal sequence of background images and wherein the perspectivetransformation computation unit is operative in real time relative to apace at which the background image sequence source provides the temporalsequence to the background generation unit.
 4. The chromakeying systemas in claim 1, further comprising a user interface for communicatingwith the background generation unit and operative to determine, forindividual virtual object locations within the background image, whethereach virtual object location is in front of the foreground subject or inback of the foreground subject and wherein the background generationunit is operative to superimpose the foreground subject over a portionof each virtual object location that is in back of the foregroundsubject and to superimpose a portion of the transformed background overeach virtual object location that is in front of the foreground subject.5. The chromakeying system according to claim 4 wherein each individualvirtual object location comprises a single pixel within the backgroundimage.
 6. The chromakeying system of claim 1, further comprising arangefinder boresighted to a camera for producing the video image of atleast a portion of the chromakeying pattern and the foreground subjectand operative to measure a range of a plurality of locations within afield of view of the camera and wherein the background generation unitis operative to merge the transformed background such that the compositeimage is generated by displaying the transformed background on locationsthat have a range that is smaller than a given range.
 7. Thechromakeying system of claim 3, wherein the coded chromakeying patterncomprises a plurality of straight lines that intersect to definevertices, and wherein the plurality of straight lines are spaced fromeach other so that for a given vertex a ratio of the distance of one ofthe lines extending from the given vertex to a first adjacent vertex tothe distance of another of the lines extending from the given vertex toa second adjacent vertex is unique.
 8. A method of compositing a virtualscene with a video image created by a video camera in which the virtualscene automatically changes perspective in relation to movement of thevideo camera creating the video image, comprising the steps of:capturinga video image of a background chromakey panel, said panel having adefined pattern comprising a plurality of straight lines that intersectto define vertices, and wherein the plurality of straight lines arespaced from each other so that for a given vertex a ratio of thedistance of one of the lines extending from the given vertex to a firstadjacent vertex to the distance of another of the lines extending fromthe given vertex to a second adjacent vertex is unique; processing thevideo image to obtain perspective information that identifies aperspective of the camera relative to the panel independently in eachvideo field of the camera; and transforming a video image of the virtualscene based on the perspective information that identifies theperspective of the camera.